The software testing phase is a critical phase in the software development process. During the software development process, the software testing phase occurs after the software has been designed, implemented in a programming language, and tested to a limited degree. During the testing phase, software testers test the software extensively to ensure that the software meets all of the requirements it is intended to meet. In order to accommodate simultaneous testing of several different software packages by several testers, multiple test machines are often implemented. Different types of software packages may need to be tested on different types of test machines, such as, for example, test machines with different hardware configurations and/or different operating systems. When a large number of software testers are required to share common resources for software testing, provisions must be made for scheduling the tests in order to efficiently manage these shared resources. The efficient management of these shared resources may also require that tests and the results of the tests be recorded so that the tests can be used repeatedly if needed and so that the results of the tests can be analyzed and subsequently used for comparison with the results of tests performed at a later time.
In an effort to maximize efficiency in the handling of test scheduling and test execution, attempts have been made to automate software testing by using a server to manage test machines and to allocate test packages among the test machines in accordance with a schedule. Sun Microsystems, Inc. has proposed an automated task-based scheduler for use with UNIX platform systems which allows users operating "client" machines to schedule tests to be executed on "target" machines. A central server receives a request from a client machine to perform a task. The server maintains information relating to all currently scheduled tasks on all target machines in a "status" database. The server maintains information relating to the expected duration of each test package and other test package attributes in a "packages" database. When the server receives a request to perform a task from a client machine, the server determines the loads on each of the target machines based on the expected duration of each test package and then schedules the task on the target machine with the least current load. A task file created at the client machine and copied to the server includes priority information relating to the task requested by the client machine. The target machine selects a task to be performed based on this priority information. Once a task is completed, the results are copied back to the server which compares them to a set of "golden results" and creates a comparison report which is mailed back to the user that requested the test.
One disadvantage of the system proposed by Sun Microsystems is that the system cannot be easily expanded. If a test machine is added to the system, the "machines" database will need to be manually updated to include information relating to the new test machine, such as its operating system and identity. Also, once a test machine is added, no provision is made for automatically rescheduling tasks taking the new test machine into consideration. This reduces the flexibility and adaptability of the system. Furthermore, if a task fails after performance of the task has begun, or if a test machine fails after tasks have been scheduled on it, all of the affected tasks will need to be rescheduled.
Another disadvantage of the system proposed by Sun Microsystems is that it is not a distributed system. The system proposed by Sun Microsystems is a local area network (LAN) having a client/server arrangement which uses scripts to communicate instructions between the users' client machines and the central server and between the central server and the target machines. No provision is made for allowing users located remotely with respect to the testing system to access the system because no communications protocol is provided for encapsulating and transporting the scripts. Also, the test system proposed by Sun Microsystems also has no provision for automatically configuring a test machine and automatically installing an operating system on a test machine. Rather, the test system proposed by Sun Microsystems only allows a user to select a test machine already having the operating system required for executing a particular test. Therefore, if a test machine having the necessary operating system for testing the software is not available, the test cannot be executed.
Accordingly, a need exists for an automated testing system which is automatically expandable to allow new test machines to be easily and automatically added to the system, and which is distributed such that the users and components of the test system may be distributed over any network, such as the Internet.